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ABSTRACT 

A common problem in the design of expert systems is the 
definition of rules from data obtained in system operation or 
simulation. While it is relatively easy to collect data and to log 
the comments of human operators engaged in experiments, 
generalizing such information to a set of rules has not previously 
been a straightforward task. This paper presents a statistical 
method for generating rule bases from numerical data, motivated 
by an example based on aircraft navigation with multiple 
sensors. The specific objective is to design an expert system 
that selects a satisfactory suite of measurements from a 
dissimilar, redundant set, given an arbitrary navigation geometry 
and possible sensor failures. This paper describes the 
systematic development of a Navigation Sensor Management 
(NSM) Expert System from Kalman Filter covariance data. The 
development method invokes two statistical techniques: Analysis 
of Variance (ANOVA) and the ID3 algorithm . The ANOVA 
technique indicates whether variations of problem parameters 
give statistically different covariance results, and the ID3 
algorithm identifies the relationships between the problem 
parameters using probabilistic knowledge extracted from a 
simulation example set. ANOVA results show that statistically 
different position accuracies are obtained when different 
navigation aids are used, the number of navigation aids is 
changed, the trajectory is varied, or the performance history is 
altered. By indicating that these four factors significantly affect 
the decision metric, an appropriate parameter framework was 
designed, and a simulation example base was created. The 
example base contained over 900 training examples from nearly 
300 simulations. The ID3 algorithm then was applied to the 
example base, yielding classification "rules” in the form of 
decision trees. The NSM expert system consists of seventeen 
decision trees that predict the performance of a specified 
integrated navigation sensor configuration. The performance of 
these decision trees was assessed on two arbitrary trajectories, 
and the performance results are presented using a predictive 
metric. The test trajectories used to evaluate the system's 
performance show that the NSM Expert adapts to new situations 
and provides reasonable estimates of sensor configuration 
performance. 

INTRODUCTION 

Knowledge acquisition is a major problem in the 
development of rule-based systems. The tools developed to date 
are not designed to extract information from data for which no 
generalizations are known a priori. Instead, these tools either 
rely on the expert to provide examples from which rules are 
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generated or try to capture the expert’s problem-solving 
methodology with interviewing techniques [1]. Unfortunately, 
it often is difficult for experts to describe their problem-solving 
methods or to detail the factors that come into play during the 
resolution of a problem. It is exactly this type of knowledge that 
is needed to design rule-based systems. 

Since the early 1970’s adaptive navigation has been viewed 
as a highly desirable candidate for development in next- 
generation aircraft [2]. It is envisioned that future aircraft will 
have multi- sensor capability for navigation tasks requiring high 
reliability, optimal performance, and increased automation. 
With multi-sensor capability, the task of sensor configuration 
selection and management will become an additional pilot 
burden. 

The performance of multi-sensor navigation systems (more 
commonly known as "integrated" or "hybrid" systems) has been 
explored since the late 1960's when results from modem control 
theory provided techniques for sensor mixing and optimal state 
estimation [3]. Hybrid systems refer to externally referenced 
navigation systems that "aid" an on-board inertial navigation 
system (INS) using an optimal state estimation mechanization. 
Hybrid navigation systems combine the high- and low- 
frequency accuracy properties of INSs and external navigation 
aids (navaids) respectively. Many radio navigation and on- 
board systems aiding INS have been modelled and their 
performance covariance results obtained [4-8]. When radio 
navigation systems are only partially operational, results show 
that improved navigation performance is obtained over that of 
the pure INS [4]. Therefore it becomes advantageous to keep 
partially operational systems as candidates for integrated sensor 
mixing purposes. 

With a large number of available navaids, choosing an 
optimal or near-optimal sensor set becomes a large combinatorial 
problem. Convergence towards an optimal sensor configuration 
requires an exhaustive computer search utilizing simulation 
results as the basis for selection. In contrast, a small number of 
available navaids reduces the decision space considerably. 
Hence, a dilemma occurs; increasing sensor capability (and thus 
reliability and performance) increases decision-making 
complexity. 

The selection of an optimal configuration requires the 
application of some decision criteria. Most often, designers 
choose between navaids based on the relative accuracies of each 
system using a hierarchical approach [9]. This approach is 
"knowledge-based" in the sense that the nominal performance of 
the systems is well-known and that this knowledge is built into 
the sensor hierarchy. The current hierarchical designs are not as 
"robust" with respect to sensor availability and performance 
changes as is necessary for future sensor management systems 
[10]. Instead, these hierarchies represent "mles-of-thumb" that 
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are useful in only the simplest cases. They do not resolve 
sensor configuration problems when more detailed information 
must be considered - for example when the number of each 
available navaid is specified, when partially operational systems 
remain viable candidates, and when trajectory effects degrade 
system performance. It becomes necessary to explore factors 
other than the performance of nominally operating navaids to 
determine how these factors affect the decision-making process, 
and to exploit the potential of hybrid systems. 

The statistical technique Analysis of Variance (ANOVA) 
[11] was used to identify the factors that cause variation in 
navigation performance. Once the important factors were 
identified, the relationships between them were determined. The 
ID3 algorithm [12,13], an inductive inference technque based on 
the probabilistic occurrence of events, was used to find these 
attribute relationships. 

The development of a navigation sensor management expert 
system using the ANOVA/ID3 technique [14] is described in 
this paper. The NSM system controls the selection of multi- 
sensor configurations. The methodology is applicable to any 
problem where the development of knowledge bases from multi- 
factor data studies is desired. 

INTEGRATED NAVIGATION SYSTEMS 

Optimal estimation techniques are used to combine inertial 
and radio navigational systems in order to provide stable 
continuous inertial navigation information [15]. The errors 
exhibited by these "hybrid" systems depend on the accuracy of 
the aiding system, and navaid accuracies are functions of many 
factors such as navaid type, number of similar navaids, and 
trajectory parameters such as distance from the navaid and 
whether the aircraft is approaching or receding from the station. 
The sensor selection criteria depend on the relative importance of 
these factors. Five external radio navigation and two on-board 
navaids were used to update a medium-accuracy (10 N. Mi/hr) 
INS. Hybrid system performance was simulated using the 
linearized inertial navigation error model and navaid 
measurement models as inputs into the optimal estimation filter. 
The hybrid errors were updated at a specified navaid fix rate. 
The systems simulated were (1) Global Positioning System 
(GPS), (2) Long-Range Navigation System (LORAN), (3) 
Tactical Navigation System (TACAN), (4) Distance Measuring 
Equipment (DME), (5) VHF Omnidirectional Range (VOR), (6) 
Doppler radar, and (7) air data sensor. The operational theory 
and the mathematical models used to simulate the navaids and 
the inertial navigation error model are discussed in detail in [14]. 

The numerically-stable discrete-time U-D implementation of 
the Kalman Filter equations was used to mix the inertial system 
and navaid information optimally, providing covariance 
estimates of the navigation errors (e.g., north/east position) 
[14,17]. Each nonlinear measurement equation was linearized 
with respect to the inertial navigation states to obtain the 
observation matrix used in the U-D measurement update 
equations. Since sensor errors were taken into consideration in 
the measurement models, the inertial error state vector was 
augmented with the sensor shaping filter dynamics (e.g., 
random bias, first-order Markov model) to formulate the hybrid 
navigation model. Additionally, the measurement noise time 
history was simulated. As the aircraft moves along its trajectory 
relative to ground-based navaid stations, the -measurement noise, 
characteristics change. Therefore an equation for a distance- or 
time-varying measurement covariance matrix was found in order 
to realistically model ground-based radio navigation systems. 
According to Ref. 17, GPS measurement noise increases in a 
similar way; as the satellite descends near the aircraft's horizon, 
the noise increases. To simulate time-varying measurement 


noise for the ground- and satellite-based navigation systems, 
each noise variance was modelled as the sum of initial and 
range-dependent variances. The latter component increases 
linearly with the square of the distance from the station or 
satellite. 

Position accuracy was selected for the rule-based system 
decision metric. Here, position accuracy is defined as the root 
sum of squares (RSS) of the north and east component errors. 
The RSS decision metric provides sufficiently consistent 
quantities to compare hybrid performances. For a detailed 
discussion of the RSS decision metric, the reader is directed to 
Ref. 14. 

HYBRID NAVIGATION SIMULATION RESULTS 

Using the RSS position error metric to measure hybrid 
system performance, the following U-D filter simulations were 
performed: 

1. Single-type hybrids: GPS, LORAN, TACAN, DME, 

VOR, Doppler Radar, or Air Data Sensor aiding an INS 

2. Number of stations used in a single-type hybrid 

3 . Multi-type hybrids: Combinations of different navaid types 

aiding an INS 

4. Aircraft trajectories simulated: High-performance, 

commercial, general aviation 

Comparisons of Single-Type Hybrid Performance 

Consider the four ground stations A, B, C, and D spatially 
oriented with respect to the high-performance, commercial, and 
general aviation trajectories in Fig. 1. The four ground stations 
are simulated as LORAN slaves, TACAN, DME, or VOR 
stations. Figure 2 shows the performance differences of 
ground-based, GPS, and on-board type hybrid systems. When 
the results from all ground station A types (LORAN, TACAN, 
DME, VOR) are compared on the high-performance trajectory, 
the relative performance from best to worst may be listed as 
follows: (1) LORAN, (2) TACAN, (3) DME, and (4) VOR. 
For example, a hybrid system utilizing LORAN slave station A 
provides better performance than a hybrid system utilizing 
TACAN A; a TACAN A hybrid in turn outperforms a DME A 
hybrid which in turn outperforms a VOR A hybrid. This pattern 
is repeated for stations B, C, and D [14]. The best hybrid 
performance was obtained from three GPS satellites aiding the 
INS. Figure 2 also shows how the performances of the 
Doppler radar hybrid and the air data sensor hybrid compare 
with the GPS and ground-based navaid hybrids. 

Referring to the LORAN results in Fig. 3, there is a 
striking variation in the performance of the individual Stations 
A-D; this figure reveals that single stations of the same type 
aiding an INS give highly variable performance results. The 
same variability in performance of the remaining ground-based 
single-station navaids was found [14], From Fig. 3, the 
variation in Station A-D's performances is attributed to the 
position of each ground station relative to the aircraft's 
trajectory. For example, LORAN Slave A gives the smallest 
position error of the four stations; referring to Fig. 1, the 
aircraft makes a close approach to Slave A on the trajectory’s 
second leg. Hence the RSS error becomes very small. These 
errors begin to increase towards the end of the trajectory leg, 
due to the increasingly uncertain north component. In contrast, 
LORAN Slaves B, C, and D are farther from the aircraft's 
trajectory. The first trajectory leg results in good relative north 
information to B, C, and D, whereas the east component 
uncertainty grows due to the lack of relative east information. 
The variations in performance observed from Stations A-D are 
due to trajectory effects; using Station B instead of A to update 
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Figure 1. Aircraft Trajectories Used in Simulations 


the INS is equivalent to using A and changing the aircraft's 
trajectory. 

Effect of Increasing the Number of Navaids in a 
Hybrid System 

Next, the effect of the number of ground stations was 
studied by simulating all possible combinations of single, 
double, and triple stations formed from stations A-D. There are 
six possible combinations of two stations and four combinations 
of three stations that may be integrated to aid the INS. These 
simulations were carried out for LORAN, TACAN, DME and 
VOR. 

Referring to the LORAN results in Fig. 4, the performance 
variation among the double station combinations and triple 
station combinations is less pronounced than the single station 
variations. The magnitude of the RSS errors decreases 
dramatically when two stations are used instead of one station. 
The RSS errors decrease further when three stations are used, 
although the magnitude differences are not as great. The reason 
why the RSS magnitudes of the double- and triple-station 
combinations are much lower is that the aircraft receives the best 
navigation information available. This also explains why there 
is much more variation in the results for the double station 
combinations than for the triple stations. Similar performance 
trends were observed for GPS, TACAN, DME, and VOR [14]. 

Effect of Trajectory on Hybrid Performance 

It already has been shown that an aircraft's trajectory 
relative to a single ground station hybrid plays an important role 
in the estimator's performance. The RSS results in Fig. 5 
illustrate the performance differences of the LORAN Slave A 
hybrid on the high-performance, commercial transport, and 
general aviation trajectories. Two parameters that contribute to 
these performance differences are distance to a station and 
heading with respect to a station. A third trajectory parameter 
that contributes to a hybrid system’s performance is the number 
of heading changes along the trajectory. The effect of heading 
changes is discussed in more detail in [14]. Trajectory factors 
affect the INS dynamics, which in turn affects the error 
estimation performance. The trajectory factors also change the 
measurement dynamics since the measurements are dependent 
on the trajectory's geometric properties and aircraft states (such 
as velocity). Tbe results in Fig. 5 clearly show that when the 
trajectory changes, the navaid selection decision most likely 
changes as well since the relative accuracies of the navaids 
change. 



o 10 20 30 40 50 60 


Time, minutes 

Figure 2. Performance of Satellite, Ground 

Station-Based and On-Board Hybrid 
Navigation Systems 


Hybrid Performance of Mixed Navaids 

Figure 6 shows various combinations of integrated navaids. 
The individual performances of LORAN Slave B, Doppler 
radar, and Air Data hybrids are shown in Fig. 6 along the high- 
performance trajectory. The LORAN/Doppler and LORAN/Air 
data hybrids also are plotted in this figure for comparison. Both 
combinations gave better results than their individual 
components operating alone. For example, the 
LORAN/Doppler combination outperformed the LORAN hybrid 
and the Doppler hybrid; similarly, the LORAN/Air Data 
combination gave better results than did the LORAN alone or the 
Air Data sensors alone. The latter combination did slightly 
better than Doppler hybrid on this trajectory after the initial 
transient period. These results show that good navigation 
performance is still obtainable when a "failed" LORAN system 
(only one slave station operational) is integrated with an on- 
board navaid such as Doppler radar or a standard equipment air 
data sensor. 
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Figure 3. Performance of Single-Station LORAN 
Navaids Aiding an INS. 


DEVELOPMENT OF A NAVIGATION SENSOR 
MANAGEMENT EXPERT SYSTEM 

This section describes a novel methodology that uses 
established statistical techniques to develop the NSM expert 
from the simulation data. The primary function of this expert 
system is to select the external navaid sensors that provide the 
smallest possible RSS position error from a large set of available 
sensors. The Analysis of Variance (ANOVA) technique [11] is 
used to identify the factors that make statistically significant 
contributions to the decision metric. Then, the ID3 algorithm 
determines the relationships between these factors [11,13]. 

Identifying Important Factors Using ANOVA 

The ANOVA technique was applied as follows: first, the 
mean value of the RSS position error and the variance for all the 
simulations were computed. The ANOVA model decomposes 
the variance into a sum of variances, each associated with a 
potentially contributing factor. Over two hundred simulations 
were performed, and the data were used in a four-factor navaid 
experiment. The goal of the experiment was to identify which 
of the factors (navaid type, number of ground stations, 
trajectory effects, performance history) and their interactions had 
statistically significant impacts on the RSS position error. The 
factor states used in the ANOVA experiment were: 
Navaids={ VOR, DME, LORAN, TACAN, GPS); Number of 
Ground Stations=(One, Two, Three); Trajectory Type={ High- 
Performance, Commercial Transport, General Aviation, from 
Fig. 1 ); Time Interval = (I, II, III, IV). Since each trajectory 
consists of four, fifteen-minute legs, the "Time Interval" factor 
refers to the RSS performance obtained within each fifteen 
minute time frame. Four single-station, six double- station, and 
four triple-station hybrids were simulated using combinations of 
Stations A-D in Fig. 1. 

The ANOVA results [14] show that three of the four factors 
are strongly significant with 99% confidence; the fourth factor, 
trajectory, was shown to be weakly significant (90% 
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Figure 4. Performance of Double-Station LORAN 
Navaids Aiding an INS 


confidence). The latter result suggested that additional 
investigation into the effect of trajectory on RSS position error is 
necessary for more specific trends to be observed. Indeed the 
term "trajectory" is extremely vague; the results from Scheffe 
comparison tests suggest that "trajectory" should be 
decomposed into attributes that describe, in better detail, what 
these effects really are. For example, some trajectory attributes 
include distance from a station, airspeed, and whether the 

aircraft is approaching or receding from the station. Scheffe 
multiple comparison tests were applied to the navaid and 
number of ground station factors to identify the specific 
differences within each groups; for example, the RSS 
performance difference between GPS and TACAN, all other 
factors being equal, was statistically significant. On the other 
hand, the RSS performance difference between LORAN and 
TACAN with all other factors being equal, was not statistically 
significant. This means that a LORAN hybrid could perform 
better or worse than a TACAN hybrid, depending on the values 
of the other factors (e.g., number of ground stations). The 
multiple comparison test results yielded the same performance 
ranking depicted in the graphical results (e.g.. Fig. 2), while 
utilizing the information content of a large number of 
independent simulations. Further investigation into the ANOVA 
interaction effects revealed that the ranking should be cautiously 
applied to single-station hybrids, since these are highly-sensitive 
to trajectory effects. The complete factor analysis results are 

given in Ref. 14. In summary, the ANOVA and Scheffe 
methods systematically identified trends in the simulation data 
without recourse to tedious graphical analysis. 

Extracting Rules Using Induction: The ID3 Algorithm 

The ID3 Algorithm uses inductive inference to extract rules 
[13] from a training set of examples. The problem space is 
described in terms of attributes, where each attribute is 
characterized by a set of values that define the possible "states." 
For example, in the previous section, the navaid type and 
number of ground stations were shown to be attributes affecting 
RSS position error. The attribute values for the factor "navaid 
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Figure 5. Comparison of RSS Results for LORAN 
Hybrids on Three Different Trajectories 


type" were {GPS, LORAN, TACAN, DME, VOR), and the 
attribute values for the factor "number of stations" were {One, 
Two, Three). Hence there is a clear connection between 
ANOVA and ID3 problem structures. ANOVA factors are ID3 
attributes, and ANOVA factor levels are ID3 attribute values. 

An important problem in designing an inductive inference 
algorithm is identifying the attributes that span the problem 
space most efficiently, so that the resulting decision tree is as 
compact as possible. The ID3 algorithm selects the most 
important attributes using an information-theoretic measure 
(ITM) that minimizes the number of tests (attribute nodes) 
necessary to classify a problem. The ID3 algorithm utilizes a 
splitting strategy [12] to decide which attribute provides the 
most information from the example set. A detailed example 
illustrating how the splitting strategy is used to construct 
classification rules is given in [14]. 

Developing the ID3 Attribute Framework Using 
ANOVA Results 

Up to three ground stations (four GPS satellites) were 
included as possible configurations. Time-weighted 
measurement effects are included in the attribute framework 
using RSS position error classification codes representing the 
hybrid's performance on a preceding trajectory leg. The 
trajectory effects were separated into the following attributes: 
geodetic distance from a ground station, line-of-sight angle from 
the station, and the direction of flight (approaching or receding) 
relative to a ground station. The distance from a ground station 
is an important attribute since the signal-to-noise ratio decreases 
as the distance to the station increases. The direction of flight 
with respect to the station influences position accuracy through 
its effect on the line-of-sight angle. The trajectory parameters 
were computed for each of the high-performance, jet transport, 
and general aviation trajectories on each trajectory leg. The 
maximum and minimum distances to the aiding station were also 
determined on each trajectory leg, in addition to the difference 
between the maximum and minimum distances. 

When more than one station was used, the attributes were 
redefined slightly. The maximum and minimum distances then 



Figure 6. Comparison of RSS Results with LORAN, 

Doppler Radar, and Air Data Sensor Hybrid 
Combinations 


referred to the closest and farthest distances computed to the 
stations. The distance difference is the algebraic difference 
between the farthest and closest distances determined on the 
trajectory leg. A similar definition was applied to the line-of- 
sight angle; from the angles computed to each station, the 
largest and smallest were selected. The ID3 algorithm’s task 
was then to determine how these attributes were related to each 
other and to the RSS performance. 

The classification scheme chosen to represent the RSS 
position error endnode in the decision trees is depicted in Table 
I. Since an approximate prediction of the RSS position error 
was of interest, it was appropriate to represent the RSS 
performance in terms of an error range. 

Table I RSS Position Error Classification 
Scheme 


Accuracy 

[High] [Medium] [Low] 


Error 

Code 

Error 

Code 

Error 

Code 

(N. Mi.) 


(N. Mi.) 


(N. Mi.) 


0.00-0.02 

c-1 

0.10-0.20 

c-6 

1. 0-1.5 

c-15 

0.02-0.04 

c-2 

0.20-0.30 

c-7 

1. 5-2.0 

c-16 

0.04-0.06 

c-3 

0.30-0.40 

c-8 

2. 0-2.5 

c-17 

0.06-0.08 

c-4 

0.40-0.50 

c-9 

2.5-3.0 

c-18 

0.08-0.10 

c-5 

0.50-0.60 

c-10 

3.0-3. 5 

c-19 



0.60-0.70 

c-11 

3. 5-4.0 

c-20 



0.70-0.80 

c-12 

4. 0-4.5 

c-21 



0.80-0.90 

c-13 

4. 5-5.0 

c-22 



0.90-1.00 

c-14 

> 5.00 

c-23 


The velocity, distance, and line-of-sight angles were 
expressed in terms of ranges instead of individual values, so that 
the expert system weights trends more heavily than specific 
examples. This renders the expert system more adaptable to 
new conditions, because matches between the actual and 
knowledge-base cases could be obtained more frequently. 
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The example set was developed using the attribute 
framework described above. The RSS position errors for each 
simulation were classified on each trajectory leg using the 
scheme in Table I. The ID3 example base was then created 
from each single-, double-, and triple- station simulation. 

NSM Decision Trees 

The NSM example set was divided into seventeen smaller 
example sets. The GPS and on-board navaid examples were 
grouped into one expert, whereas the ground-based navaid 
examples were divided according to navaid type and time (15- 
minute intervals). The ID3 algorithm constructed decision trees 
for each of the seventeen small expert systems that comprise the 
larger NSM Expert. The breakdown of the NSM Expert into 
smaller systems provides greater manageability of the training 
example base. The total number of examples used to develop 
the NSM Expert System was nine hundred and thirty-two. In 
total, two hundred and sixty Kalman Filter covariance 
simulations were performed to formulate the complete NSM 
example set. An additional thirty-seven simulations were 
performed to obtain a decision tree to estimate RSS performance 
when different navaid types are combined. The NSM expert 
system prompts the user for a set of flight conditions 
commensurate with the attribute/value lists used in the example 
set, and the resulting RSS classification code is returned to the 
user from the decision tree. 

A typical decision tree obtained for the ground-based 
navaids is exemplified by the TACAN results. Figure 7 
presents the decision trees for single-, double-, and triple-station 
combinations on the first fifteen-minute trajectory leg. Here, the 
majority of the testing nodes are trajectory parameters (distance, 
LOS angle, direction of flight with respect to the station(s)). 
The top or root node in Fig. 7 is the aircraft's direction of flight. 
This is expected because the distance and LOS angle attributes 
are dependent on directional motion. Distance, LOS angle, and 
groundspeed are results of the aircraft’s motion, and hence, 
represent more specific problem parameters; therefore it is 
expected that these parameters appear at a lower depth in the 


decision tree. Figure 7 also shows that distance, ground 
velocity, LOS angle, and hybrid performance history are 
significant factors that enable a prediction of the RSS error to be 
made. The RSS classification results verify that the closer the 
aircraft is to a station(s), the smaller is the RSS error; other 
results show that the larger is the LOS angle, the smaller is the 
RSS error [14]. 

The expected performance of the GPS system on each 
trajectory leg is shown in Fig. 8. Note that the aircraft's 
groundspeed plays an important role in the GPS hybrid's 
performance. Velocity affects the measurement dynamics 
(history) and is therefore classified as a trajectory effect. From 
Fig. 8, the two-satellite hybrids are more sensitive to these 
velocity effects than are the three- and four-satellite hybrids. 

Finally, the decision tree showing what position error range 
is expected when different navaid types are integrated in a 
hybrid system is presented in Fig. 9. Note that the decision tree 
is not specified for a given trajectory leg. The RSS position 
errors for these simulations were averaged over the entire flight 
time for the high-performance trajectory. The tree is organized in 

terms of the navigation method used: (1) Distance- Velocity (p-V), (2) Bearing- 
Velocity (0-V), (3) Distance-Bearing (p-0), (4) Distance-Distaice (p-p), (5) 
Bearing-Bearing (0-0), and (6) Vekxiy-Vekxhy (V-V). These results show 
that LQRAN is abetter distance-measuring navaid than DME and 
that Doppler Radar is a better velocity-measuring system than 

the Air Data Sensor when p-V navigation is used. The p-0 
results show that it is possible to obtain performance when 
LORAN and VOR are used. The LORAN/DME hybrid gives 
better results than two DME stations but worse performance 
than two LORAN stations. By far the worst results are 
obtained using two VOR stations. As discussed before, the 
VOR system is the least accurate measurement device of the 
seven systems studied, which greatly affects 1NS-VOR hybrid 
results. 



Figure 7. Decision Trees Predicting RSS Position Error Range for an INS 
Aided by TACAN During the First 15 Minutes of Flight 
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Figure 8 Decision Tree Predicting RSS Performance 
for an INS Aided by GPS 



Figure 9 Decision Tree Predicting RSS Performance When 

Different Navaid Combinations are Used to Aid an INS 


PERFORMANCE RESULTS OF NSM EXPERT 
SYSTEM 

It is important to quantify the NSM Expert's performance 
for several test scenarios, in terms of how well it predicts a 
given hybrid’s RSS position error. It is also important to gain 
insight into the factors that affect the system’s performance, so 
that these factors can be exploited in future system development. 


Two high-performance trajectories were used in the 
performance evaluation of the NSM Expert. The two 
trajectories each consist of four fifteen-minute legs. Trajectory 
#2's flight pattern was in a counter-clockwise direction, whereas 
clockwise flight patterns were used to develop the NSM Expert 
(Fig. 1). Additionally, the takeoff point on Trajectory #2 was 
five degrees farther north than the training trajectories’ takeoff 
points. These trajectory differences change the measurement 
and INS dynamics, and hence the hybrid performance. 
Trajectory #2 was designed this way intentionally, so that the 
NSM Expert System's adaptability could be determined. 

Single-, double-, and triple-station combination hybrids 
were simulated on each test trajectory for each of the DME, 
VOR, TACAN, and LORAN systems. The combinations were 
formed using four ground stations located as in Fig. 1 with 
respect to each other. Additionally, two-, three-, and four- 
satellite hybrids were simulated on the test trajectories, as were 


Doppler Radar and Air Data sensor hybrids. In total, sixty 
covariance simulations were performed for the two test 
trajectories. 

Test Trajectory Data Preparation, Performance 
Metrics, and Results 

The performance results for each of the sixty simulations 
were classified on each trajectory leg according to the scheme in 
Table I. The total number of matches was counted on each leg 
of each test trajectory for the seven navaid types studied. A 
match was declared between the actual and predicted RSS 
classification if and only if the RSS classification codes differed 
by one or less. For example, if the NSM Expert predicted an 
RSS classification code of 6 whereas the covariance results 
determined a performance of Class 7, a match was declared. A 
match would also have been declared if the actual performance 
was Class 5. Since the NSM Expert is only expected to estimate 
a hybrid’s performance, it is allowed some room for error. 

In total, the NSM Expert System was run four hundred and 
eighty-eight times in order to determine the number of matches 
for each system on the test trajectories. Figure 10 shows the 
NSM Expert’s performance in predicting the RSS position error 
for each hybrid configuration. The predictive performance 
metric for each navaid is defined as the percentage of number of 
matches obtained from the total number of combinations tested 
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for that navaid. The matches on all four trajectory legs are 
reflected in this figure. 

The NSM Expert performed very well on the two test 
trajectories. Figure 10 shows that the NSM Expert correctly 
predicts the RSS position error better than 70% of the time on 
test Trajectory #1. The system required only the trajectory 
information and its knowledge of hybrid system performance to 
make these predictions. However, its predictive capability on 
test Trajectory #2 is slightly worse for the LORAN hybrids, 
considerably worse for the VOR and Air Data sensor hybrids, 
and identical for the remaining configurations. Hence, the 
results from Trajectory #2 suggest that additional investigation 
into trajectory effects on VOR's and Air Data Sensor's 
performance may be necessary. 



RADAR SBSBOR 

EXPERT SYSTEMS 


The results in Fig. 10 are truly encouraging for designers of 
expert systems. We have shown that an expert system can be 
designed from data, and that good results are obtainable even 
from relatively small training sets. Recall that the total number 
of examples used to obtain the NSM decision trees was slightly 
less than one thousand. 

CONCLUSIONS 

The performances of seven navigation systems aiding a 
medium-accuracy INS were investigated using Kalman Filter 
covariance analyses. Hybrid performance decisions were based 
on the RSS position error history metric. A NSM Expert was 
designed from covariance simulation data using a systematic 
method comprised of the two statistical techniques, the Analysis 
of Variance (ANOVA) method and the ID3 algorithm. 

ANOVA results show that statistically different position 
accuracies are obtained when different navaids are used, the 
number of radio navigation ground stations or GPS satellites 
used to aid the INS is varied, the aircraft’s trajectory is varied, 
and the performance history is varied. By indicating that these 
four factors significantly affect the decision metric, an 
appropriate parameter framework was designed, and a 
simulation example base was created. 


The example base was composed of over nine hundred 
training examples from nearly three hundred simulations. The 
example base was divided into seventeen smaller groups to 
enhance manageability. The ID3 algorithm then was used to 
determine the NSM Expert’s classification "rules” in the form of 
decision trees. The performances of these decision trees were 
assessed on two arbitrary trajectories, by counting the number 
of times the rules correctly predicted the RSS position accuracy. 
These performance results then were presented using a 
predictive metric. 

The ANOVA/ID3 method was very effective for the 
systematic development of the NSM Expert using simulation 
data. Results show that the NSM Expert can predict the RSS 
position accuracy between 65 and 100% of the time for a 
specified navaid configuration and aircraft trajectory. The test 
trajectories used to evaluate the system's performance show that 
the NSM Expert adapts to new situations and provides 
reasonable estimates of the expected hybrid performance. The 
system's good performance with relatively few examples clearly 
shows how the ID3 algorithm maximizes the information 
content contained in the example base. The performance results 
strongly suggest that operational systems can be designed from 
simulation or experimental data using the ANOVA/ID3 method 
for knowledge acquisition. The systematic nature of the method 
makes it a useful tool for expert system designers. 

Other aerospace applications that are good candidates for 
the ANOVA/ID3 method are air combat pilot strategies from 
simulation or flight test data and air traffic control solutions to 
multi-configuration problems. The expert system design 
methodology also is pertinent to problems such as nuclear 
reactor control strategies, chemical process control strategies, 
automated highway driving, and robotics applications. In each 
case simulation or operational experiments may be executed for 
the systematic development of an expert system advisor. 

ACKNOWLEDGEMENTS 

This project was sponsored by the U.S. Army Research Office 
under Contract No. DAAG29-84-K-0048, and supported by 
NASA and the FAA under Grant No. NGL31-001-252. 

REFERENCES 

[1] G. Kahn et al, "MORE: An Intelligent Knowledge 
Acquisition Tool", in Proceedings of the Ninth 
International Conference on Artificial Intelligence , Los 
Angeles CA, 1985. 

[2] F.G. Unger and R.S. Sindling, "Systems Tasks for 
Advanced Aircraft NavigationSystems", in Computers in 
the Guidance and Control of Aerospace Vehicles, 
AGARD-AG-158, February 1972. 

[3] A. Gelb and A. Sutherland Jr., "Software Advances in 
Aided Inertial Navigation Systems", Navigation , The 
Institute of Navigation, Washington DC, Vol. 17, No.4, 
Winter 1970-71, pp 358-369. 

[4] J. Richman and B. Friedland, "Design of Optimum Mixer- 
Filter for Aircraft Navigation Systems", in Proceedings of 
the IEEE 1967 National Aerospace and Electronics 
Conference (NAECON) 1967, pp.429-438. 

[5] W. Zimmerman, "Optimum Integration of Aircraft 
Navigation Systems", in IEEE Transactions on Aerospace 
and Electronics Systems , Vol. AES-5, No.5, September 
1969, pp. 737-747. 


412 



[6] J. W. Burrows, "Combined Inertial-ILS Aircraft 
Navigation Systems", Journal of Aircraft, Vol. 8, No. 6, 
June 1971, pp.439-443. 

[7] D. B. Cox Jr., "Integration of GPS with Inertial 
Navigation Systems", in Global Positioning System Vol. 
II , papers published in Navigation, The Institute of 
Navigation, Washington DC, 1984, pp. 144-153. 

[8] E. J. Rose et al . , A Cost! Performance Analysis of Hybrid 
Inertial! Externally Referenced Positioning/Orientation 
Systems , Report ETL-R-086, U.S. Army Engineer 
Topographic Laboratories, September 1985. 

[9] A.D. Pisaro and H.L. Jones, "An Expert System 
Approach to Adaptive Tactical Navigation", in Proceedings 
of the First Conference on Artificial Intelligence 
Applications, IEEE Computer Society, 1982, pp. 460- 
464. 

[10] S. Berning, D. P. Glasson and J. L. Pomarede, 
"Knowledge Engineering for the Adaptive Tactical 
Navigator", in Proceedings of the IEEE National 
Aerospace and Electronics Conference (NAECON), 
Dayton, OH, May 1988, pp. 1266-1273. 

[11] G. E. P. Box, W. G. Hunter and J. S. Hunter, Statistics 
for Experiments : An Introduction to Design , Data Analysis 
and Model Building , John Wiley & Sons, New York, 
1978, Chapter 6. 

[12] Expert Ease Manual , Human Edge Software, Intelligent 
Terminals Ltd., Palo Alto, CA, 1983, 

[13] J. R. Quinlan, "Discovering Rules by Induction From 
Large Collections of Examples", in Expert Systems in the 
Micro Electronic Age , D. Michie, Editor, Edinburgh 
University Press, 1979, pp. 169-201. 

[14] B. L. Belkin, Cooperative Rule-Based Systems for 
Aircraft Navigation and Control , M.S.E. Thesis, Report 
1856T, Department of Mechanical and Aerospace 
Engineering, Princeton University, June 1989. 

[15] P. S. Maybeck, Stochastic Models, Estimation and 
Control, Vol. 1, Academic Press, New York, 1979. 

[16] R. F. Stengel, Stochastic Optimal Control - Theory and 
Application , J. Wiley & Sons, New York, 1986. 

[17] M. Kayton and W. R. Fried, Avionics Navigation 
Systems , John Wiley & Sons, New York, 1969. 


413 


